Adaptive Control of a Partially Observed Controlled Markov Chain

We consider an adaptive i~nite state controlled Maxkov chain with partial state information, motivated by a class of replacement problems. We present parameter estimation techniques based on the information ava~lable after actions that reset the state to a known v~ue are taken. We prove that the parameter estimates converge w.p.1 to the true (unknown) parameter, under the feedback structure induced by a certainty equivalent adaptive policy. We also show that the adaptive policy is self-optimizing, in a long-run average sense, for any (measurable) sequence of parameter estimates converging w.p.1 to the true parameter. * This work was supported in part by the Texas Advanced TechnoloLrD ' Program under Grant No. 003658093, in part by the Air Force OfFice of Scientific Research under Grants AFOSRo91-0033, F49620-92-J-0045, and F49620-92-J-0083, and in part by the Nation ~I Science Foundation under Grant CDR-8803012. t Systems and Industrial Engineering Department, The University of Arizona, Tucson, Arizona 85721. Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas 78712-I084. § Department of Electrical Engineering and Systems Research Center, The University of Maryland, College Park, Maryland 20742.